Repository navigation
fix(writer): text is a vortex.varbinview unless a scheme is smaller (#583) - #677
Merged
Merged
Conversation
…as Rust's canonical form (#583) Rust's canonical text array is vortex.varbinview and the baseline every string scheme must beat; VarBinScheme re-encodes binary only. The Utf8 competition now scores against the views' size, leaves vortex.varbin and vortex.varbinview out as candidates for Utf8 (varbinview for binary too), and falls back to varbinview. The node carries is_constant, null_count and the size of its views, and an empty array's views buffer is empty, which Rust checks against the row count. The string zone tables now match vortex-jni, bar one whose three bounds Rust's trained FSST table compresses. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV
dfa1
force-pushed
the
feat/583-utf8-varbinview
branch
from
November 6, 2026 20:35
b8b840b to
358dda5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #676.
vortex.varbinview, the baseline every scheme must beat;VarBinSchemeis binary-only. The Utf8 competition scores against view bytes, excludes varbin for Utf8, and falls back to varbinview.is_constant,null_count, view size.high, whose 3 bounds Rust's trained FSST table compresses.Not done:
is_sortedon varbin/FSST offsets and FSST lengths shape — depend on the trained symbol table.🤖 Generated with Claude Code
https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV